<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="../../aosa.css" type="text/css">
    <title>500 Lines or Less: A Simple Web Server</title>
  </head>
  <body>

    <div class="titlebox">
      <h1>500 Lines or Less<br>A Simple Web Server</h1>
      <p class="author">Greg Wilson</p>
    </div>

    <p><em><a href="https://twitter.com/gvwilson">Greg Wilson</a> is the founder of Software Carpentry, a crash course in computing skills for scientists and engineers. He has worked for 30 years in both industry and academia, and is the author or editor of several books on computing, including the 2008 Jolt Award winner <em>Beautiful Code</em> and the first two volumes of <em>The Architecture of Open Source Applications</em>. Greg received a PhD in Computer Science from the University of Edinburgh in 1993.</em></p>

<h2 id="introduction">Introduction</h2>

<p>The web has changed society in countless ways over the last two decades, but its core has changed very little. Most systems still follow the rules that Tim Berners-Lee laid out a quarter of a century ago. In particular, most web servers still handle the same kinds of messages they did then, in the same way.</p>

<p>This chapter will explore how they do that. At the same time, it will explore how developers can create software systems that don't need to be rewritten in order to add new features.</p>

<h2 id="background">Background</h2>

<p>Pretty much every program on the web runs on a family of communication standards called Internet Protocol (IP). The member of that family which concerns us is the Transmission Control Protocol (TCP/IP), which makes communication between computers look like reading and writing files.</p>

<p>Programs using IP communicate through sockets. Each socket is one end of a point-to-point communication channel, just like a phone is one end of a phone call. A socket consists of an IP address that identifies a particular machine and a port number on that machine. The IP address consists of four 8-bit numbers, such as <code>174.136.14.108</code>; the Domain Name System (DNS) matches these numbers to symbolic names like <code>aosabook.org</code> that are easier for human beings to remember.</p>

<p>A port number is a number in the range 0-65535 that uniquely identifies the socket on the host machine. (If an IP address is like a company's phone number, then a port number is like an extension.) Ports 0-1023 are reserved for the operating system's use; anyone else can use the remaining ports.</p>

<p>The Hypertext Transfer Protocol (HTTP) describes one way that programs can exchange data over IP. HTTP is deliberately simple: the client sends a request specifying what it wants over a socket connection, and the server sends some data in response (<a href="#figure-22.1">Figure 22.1</a>.) The data may be copied from a file on disk, generated dynamically by a program, or some mix of the two.</p>

<div class="center figure">
<a name="figure-22.1"></a><img src="web-server-images/http-cycle.png" alt="Figure 22.1 - The HTTP Cycle" title="Figure 22.1 - The HTTP Cycle" />
</div>

<p class="center figcaption">
<small>Figure 22.1 - The HTTP Cycle</small>
</p>

<p>The most important thing about an HTTP request is that it's just text: any program that wants to can create one or parse one. In order to be understood, though, that text must have the parts shown in <a href="#figure-22.2">Figure 22.2</a>.</p>

<div class="center figure">
<a name="figure-22.2"></a><img src="web-server-images/http-request.png" alt="Figure 22.2 - An HTTP Request" title="Figure 22.2 - An HTTP Request" />
</div>

<p class="center figcaption">
<small>Figure 22.2 - An HTTP Request</small>
</p>

<p>The HTTP method is almost always either &quot;GET&quot; (to fetch information) or &quot;POST&quot; (to submit form data or upload files). The URL specifies what the client wants; it is often a path to a file on disk, such as <code>/research/experiments.html</code>, but (and this is the crucial part) it's completely up to the server to decide what to do with it. The HTTP version is usually &quot;HTTP/1.0&quot; or &quot;HTTP/1.1&quot;; the differences between the two don't matter to us.</p>

<p>HTTP headers are key/value pairs like the three shown below:</p>

<pre><code>Accept: text/html
Accept-Language: en, fr
If-Modified-Since: 16-May-2005</code></pre>

<p>Unlike the keys in hash tables, keys may appear any number of times in HTTP headers. This allows a request to do things like specify that it's willing to accept several types of content.</p>

<p>Finally, the body of the request is any extra data associated with the request. This is used when submitting data via web forms, when uploading files, and so on. There must be a blank line between the last header and the start of the body to signal the end of the headers.</p>

<p>One header, called <code>Content-Length</code>, tells the server how many bytes to expect to read in the body of the request.</p>

<p>HTTP responses are formatted like HTTP requests (<a href="#figure-22.3">Figure 22.3</a>):</p>

<div class="center figure">
<a name="figure-22.3"></a><img src="web-server-images/http-response.png" alt="Figure 22.3 - An HTTP Response" title="Figure 22.3 - An HTTP Response" />
</div>

<p class="center figcaption">
<small>Figure 22.3 - An HTTP Response</small>
</p>

<p>The version, headers, and body have the same form and meaning. The status code is a number indicating what happened when the request was processed: 200 means &quot;everything worked&quot;, 404 means &quot;not found&quot;, and other codes have other meanings. The status phrase repeats that information in a human-readable phrase like &quot;OK&quot; or &quot;not found&quot;.</p>

<p>For the purposes of this chapter there are only two other things we need to know about HTTP.</p>

<p>The first is that it is <em>stateless</em>: each request is handled on its own, and the server doesn't remember anything between one request and the next. If an application wants to keep track of something like a user's identity, it must do so itself.</p>

<p>The usual way to do this is with a cookie, which is a short character string that the server sends to the client, and the client later returns to the server. When a user performs some function that requires state to be saved across several requests, the server creates a new cookie, stores it in a database, and sends it to her browser. Each time her browser sends the cookie back, the server uses it to look up information about what the user is doing.</p>

<p>The second thing we need to know about HTTP is that a URL can be supplemented with parameters to provide even more information. For example, if we're using a search engine, we have to specify what our search terms are. We could add these to the path in the URL, but what we should do is add parameters to the URL. We do this by adding '?' to the URL followed by 'key=value' pairs separated by '&amp;'. For example, the URL <code>http://www.google.ca?q=Python</code> asks Google to search for pages related to Python: the key is the letter 'q', and the value is 'Python'. The longer query <code>http://www.google.ca/search?q=Python&amp;amp;client=Firefox</code> tells Google that we're using Firefox, and so on. We can pass whatever parameters we want, but again, it's up to the application running on the web site to decide which ones to pay attention to, and how to interpret them.</p>

<p>Of course, if '?' and '&amp;' are special characters, there must be a way to escape them, just as there must be a way to put a double quote character inside a character string delimited by double quotes. The URL encoding standard represents special characters using '%' followed by a 2-digit code, and replaces spaces with the '+' character. Thus, to search Google for &quot;grade = A+&quot; (with the spaces), we would use the URL <code>http://www.google.ca/search?q=grade+%3D+A%2B</code>.</p>

<p>Opening sockets, constructing HTTP requests, and parsing responses is tedious, so most people use libraries to do most of the work. Python comes with such a library called <code>urllib2</code> (because it's a replacement for an earlier library called <code>urllib</code>), but it exposes a lot of plumbing that most people never want to care about. The <a href="https://pypi.python.org/pypi/requests">Requests</a> library is an easier-to-use alternative to <code>urllib2</code>. Here's an example that uses it to download a page from the AOSA book site:</p>

<pre class="sourceCode python"><code class="sourceCode python"><span class="ch">import</span> requests
response = requests.get(<span class="st">&#39;http://aosabook.org/en/500L/web-server/testpage.html&#39;</span>)
<span class="dt">print</span> <span class="st">&#39;status code:&#39;</span>, response.status_code
<span class="dt">print</span> <span class="st">&#39;content length:&#39;</span>, response.headers[<span class="st">&#39;content-length&#39;</span>]
<span class="dt">print</span> response.text</code></pre>

<pre><code>status code: 200
content length: 61
&lt;html&gt;
  &lt;body&gt;
    &lt;p&gt;Test page.&lt;/p&gt;
  &lt;/body&gt;
&lt;/html&gt;</code></pre>

<p><code>request.get</code> sends an HTTP GET request to a server and returns an object containing the response. That object's <code>status_code</code> member is the response's status code; its <code>content_length</code> member is the number of bytes in the response data, and <code>text</code> is the actual data (in this case, an HTML page).</p>

<h2 id="hello-web">Hello, Web</h2>

<p>We're now ready to write our first simple web server. The basic idea is simple:</p>

<ol style="list-style-type: decimal">
<li>Wait for someone to connect to our server and send an HTTP request;</li>
<li>parse that request;</li>
<li>figure out what it's asking for;</li>
<li>fetch that data (or generate it dynamically);</li>
<li>format the data as HTML; and</li>
<li>send it back.</li>
</ol>

<p>Steps 1, 2, and 6 are the same from one application to another, so the Python standard library has a module called <code>BaseHTTPServer</code> that does those for us. We just have to take care of steps 3-5, which we do in the little program below:</p>

<pre class="sourceCode python"><code class="sourceCode python"><span class="ch">import</span> BaseHTTPServer

<span class="kw">class</span> RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
    <span class="co">&#39;&#39;&#39;Handle HTTP requests by returning a fixed &#39;page&#39;.&#39;&#39;&#39;</span>

    <span class="co"># Page to send back.</span>
    Page = <span class="st">&#39;&#39;&#39;\</span>
<span class="st">&lt;html&gt;</span>
<span class="st">&lt;body&gt;</span>
<span class="st">&lt;p&gt;Hello, web!&lt;/p&gt;</span>
<span class="st">&lt;/body&gt;</span>
<span class="st">&lt;/html&gt;</span>
<span class="st">&#39;&#39;&#39;</span>

    <span class="co"># Handle a GET request.</span>
    <span class="kw">def</span> do_GET(<span class="ot">self</span>):
        <span class="ot">self</span>.send_response(<span class="dv">200</span>)
        <span class="ot">self</span>.send_header(<span class="st">&quot;Content-Type&quot;</span>, <span class="st">&quot;text/html&quot;</span>)
        <span class="ot">self</span>.send_header(<span class="st">&quot;Content-Length&quot;</span>, <span class="dt">str</span>(<span class="dt">len</span>(<span class="ot">self</span>.Page)))
        <span class="ot">self</span>.end_headers()
        <span class="ot">self</span>.wfile.write(<span class="ot">self</span>.Page)

<span class="co">#----------------------------------------------------------------------</span>

<span class="kw">if</span> <span class="ot">__name__</span> == <span class="st">&#39;__main__&#39;</span>:
    serverAddress = (<span class="st">&#39;&#39;</span>, <span class="dv">8080</span>)
    server = BaseHTTPServer.HTTPServer(serverAddress, RequestHandler)
    server.serve_forever()</code></pre>

<p>The library's <code>BaseHTTPRequestHandler</code> class takes care of parsing the incoming HTTP request and deciding what method it contains. If the method is GET, the class calls a method named <code>do_GET</code>. Our class <code>RequestHandler</code> overrides this method to dynamically generate a simple page: the text is stored in the class-level variable <code>Page</code>, which we send back to the client after sending a 200 response code, a <code>Content-Type</code> header telling the client to interpret our data as HTML, and the page's length. (The <code>end_headers</code> method call inserts the blank line that separates our headers from the page itself.)</p>

<p>But <code>RequestHandler</code> isn't the whole story: we still need the last three lines to actually start a server running. The first of these lines defines the server's address as a tuple: the empty string means &quot;run on the current machine&quot;, and 8080 is the port. We then create an instance of <code>BaseHTTPServer.HTTPServer</code> with that address and the name of our request handler class as parameters, then ask it to run forever (which in practice means until we kill it with Control-C).</p>

<p>If we run this program from the command line, it doesn't display anything:</p>

<pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="kw">python</span> server.py</code></pre>

<p>If we then go to <code>http://localhost:8080</code> with our browser, though, we get this in our browser:</p>

<pre><code>Hello, web!</code></pre>

<p>and this in our shell:</p>

<pre><code>127.0.0.1 - - [24/Feb/2014 10:26:28] &quot;GET / HTTP/1.1&quot; 200 -
127.0.0.1 - - [24/Feb/2014 10:26:28] &quot;GET /favicon.ico HTTP/1.1&quot; 200 -</code></pre>

<p>The first line is straightforward: since we didn't ask for a particular file, our browser has asked for '/' (the root directory of whatever the server is serving). The second line appears because our browser automatically sends a second request for an image file called <code>/favicon.ico</code>, which it will display as an icon in the address bar if it exists.</p>

<h2 id="displaying-values">Displaying Values</h2>

<p>Let's modify our web server to display some of the values included in the HTTP request. (We'll do this pretty frequently when debugging, so we might as well get some practice.) To keep our code clean, we'll separate creating the page from sending it:</p>

<pre class="sourceCode python"><code class="sourceCode python"><span class="kw">class</span> RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):

    <span class="co"># ...page template...</span>

    <span class="kw">def</span> do_GET(<span class="ot">self</span>):
        page = <span class="ot">self</span>.create_page()
        <span class="ot">self</span>.send_page(page)

    <span class="kw">def</span> create_page(<span class="ot">self</span>):
        <span class="co"># ...fill in...</span>

    <span class="kw">def</span> send_page(<span class="ot">self</span>, page):
        <span class="co"># ...fill in...</span></code></pre>

<p><code>send_page</code> is pretty much what we had before:</p>

<pre class="sourceCode python"><code class="sourceCode python">    <span class="kw">def</span> send_page(<span class="ot">self</span>, page):
        <span class="ot">self</span>.send_response(<span class="dv">200</span>)
        <span class="ot">self</span>.send_header(<span class="st">&quot;Content-type&quot;</span>, <span class="st">&quot;text/html&quot;</span>)
        <span class="ot">self</span>.send_header(<span class="st">&quot;Content-Length&quot;</span>, <span class="dt">str</span>(<span class="dt">len</span>(page)))
        <span class="ot">self</span>.end_headers()
        <span class="ot">self</span>.wfile.write(page)</code></pre>

<p>The template for the page we want to display is just a string containing an HTML table with some formatting placeholders:</p>

<pre class="sourceCode python"><code class="sourceCode python">    Page = <span class="st">&#39;&#39;&#39;\</span>
<span class="st">&lt;html&gt;</span>
<span class="st">&lt;body&gt;</span>
<span class="st">&lt;table&gt;</span>
<span class="st">&lt;tr&gt;  &lt;td&gt;Header&lt;/td&gt;         &lt;td&gt;Value&lt;/td&gt;          &lt;/tr&gt;</span>
<span class="st">&lt;tr&gt;  &lt;td&gt;Date and time&lt;/td&gt;  &lt;td&gt;</span><span class="ot">{date_time}</span><span class="st">&lt;/td&gt;    &lt;/tr&gt;</span>
<span class="st">&lt;tr&gt;  &lt;td&gt;Client host&lt;/td&gt;    &lt;td&gt;</span><span class="ot">{client_host}</span><span class="st">&lt;/td&gt;  &lt;/tr&gt;</span>
<span class="st">&lt;tr&gt;  &lt;td&gt;Client port&lt;/td&gt;    &lt;td&gt;</span><span class="ot">{client_port}</span><span class="st">s&lt;/td&gt; &lt;/tr&gt;</span>
<span class="st">&lt;tr&gt;  &lt;td&gt;Command&lt;/td&gt;        &lt;td&gt;</span><span class="ot">{command}</span><span class="st">&lt;/td&gt;      &lt;/tr&gt;</span>
<span class="st">&lt;tr&gt;  &lt;td&gt;Path&lt;/td&gt;           &lt;td&gt;</span><span class="ot">{path}</span><span class="st">&lt;/td&gt;         &lt;/tr&gt;</span>
<span class="st">&lt;/table&gt;</span>
<span class="st">&lt;/body&gt;</span>
<span class="st">&lt;/html&gt;</span>
<span class="st">&#39;&#39;&#39;</span></code></pre>

<p>and the method that fills this in is:</p>

<pre class="sourceCode python"><code class="sourceCode python">    <span class="kw">def</span> create_page(<span class="ot">self</span>):
        values = {
            <span class="st">&#39;date_time&#39;</span>   : <span class="ot">self</span>.date_time_string(),
            <span class="co">&#39;client_host&#39;</span> : <span class="ot">self</span>.client_address[<span class="dv">0</span>],
            <span class="co">&#39;client_port&#39;</span> : <span class="ot">self</span>.client_address[<span class="dv">1</span>],
            <span class="co">&#39;command&#39;</span>     : <span class="ot">self</span>.command,
            <span class="co">&#39;path&#39;</span>        : <span class="ot">self</span>.path
        }
        page = <span class="ot">self</span>.Page.<span class="dt">format</span>(**values)
        <span class="kw">return</span> page</code></pre>

<p>The main body of the program is unchanged: as before, it creates an instance of the <code>HTTPServer</code> class with an address and this request handler as parameters, then serves requests forever. If we run it and send a request from a browser for <code>http://localhost:8080/something.html</code>, we get:</p>

<pre><code>  Date and time  Mon, 24 Feb 2014 17:17:12 GMT
  Client host    127.0.0.1
  Client port    54548
  Command        GET
  Path           /something.html</code></pre>

<p>Notice that we do <em>not</em> get a 404 error, even though the page <code>something.html</code> doesn't exist as a file on disk. That's because a web server is just a program, and can do whatever it wants when it gets a request: send back the file named in the previous request, serve up a Wikipedia page chosen at random, or whatever else we program it to.</p>

<h2 id="serving-static-pages">Serving Static Pages</h2>

<p>The obvious next step is to start serving pages from the disk instead of generating them on the fly. We'll start by rewriting <code>do_GET</code>:</p>

<pre class="sourceCode python"><code class="sourceCode python">    <span class="kw">def</span> do_GET(<span class="ot">self</span>):
        <span class="kw">try</span>:

            <span class="co"># Figure out what exactly is being requested.</span>
            full_path = os.getcwd() + <span class="ot">self</span>.path

            <span class="co"># It doesn&#39;t exist...</span>
            <span class="kw">if</span> not os.path.exists(full_path):
                <span class="kw">raise</span> ServerException(<span class="st">&quot;&#39;</span><span class="ot">{0}</span><span class="st">&#39; not found&quot;</span>.<span class="dt">format</span>(<span class="ot">self</span>.path))

            <span class="co"># ...it&#39;s a file...</span>
            <span class="kw">elif</span> os.path.isfile(full_path):
                <span class="ot">self</span>.handle_file(full_path)

            <span class="co"># ...it&#39;s something we don&#39;t handle.</span>
            <span class="kw">else</span>:
                <span class="kw">raise</span> ServerException(<span class="st">&quot;Unknown object &#39;</span><span class="ot">{0}</span><span class="st">&#39;&quot;</span>.<span class="dt">format</span>(<span class="ot">self</span>.path))

        <span class="co"># Handle errors.</span>
        <span class="kw">except</span> <span class="ot">Exception</span> <span class="ch">as</span> msg:
            <span class="ot">self</span>.handle_error(msg)</code></pre>

<p>This method assumes that it's allowed to serve any files in or below the directory that the web server is running in (which it gets using <code>os.getcwd</code>). It combines this with the path provided in the URL (which the library automatically puts in <code>self.path</code>, and which always starts with a leading '/') to get the path to the file the user wants.</p>

<p>If that doesn't exist, or if it isn't a file, the method reports an error by raising and catching an exception. If the path matches a file, on the other hand, it calls a helper method named <code>handle_file</code> to read and return the contents. This method just reads the file and uses our existing <code>send_content</code> to send it back to the client:</p>

<pre class="sourceCode python"><code class="sourceCode python">    <span class="kw">def</span> handle_file(<span class="ot">self</span>, full_path):
        <span class="kw">try</span>:
            <span class="kw">with</span> <span class="dt">open</span>(full_path, <span class="st">&#39;rb&#39;</span>) <span class="ch">as</span> reader:
                content = reader.read()
            <span class="ot">self</span>.send_content(content)
        <span class="kw">except</span> <span class="ot">IOError</span> <span class="ch">as</span> msg:
            msg = <span class="st">&quot;&#39;</span><span class="ot">{0}</span><span class="st">&#39; cannot be read: </span><span class="ot">{1}</span><span class="st">&quot;</span>.<span class="dt">format</span>(<span class="ot">self</span>.path, msg)
            <span class="ot">self</span>.handle_error(msg)</code></pre>

<p>Note that we open the file in binary mode—the 'b' in 'rb'—so that Python won't try to &quot;help&quot; us by altering byte sequences that look like a Windows line ending. Note also that reading the whole file into memory when serving it is a bad idea in real life, where the file might be several gigabytes of video data. Handling that situation is outside the scope of this chapter.</p>

<p>To finish off this class, we need to write the error handling method and the template for the error reporting page:</p>

<pre class="sourceCode python"><code class="sourceCode python">    Error_Page = <span class="st">&quot;&quot;&quot;\</span>
<span class="st">        &lt;html&gt;</span>
<span class="st">        &lt;body&gt;</span>
<span class="st">        &lt;h1&gt;Error accessing </span><span class="ot">{path}</span><span class="st">&lt;/h1&gt;</span>
<span class="st">        &lt;p&gt;</span><span class="ot">{msg}</span><span class="st">&lt;/p&gt;</span>
<span class="st">        &lt;/body&gt;</span>
<span class="st">        &lt;/html&gt;</span>
<span class="st">        &quot;&quot;&quot;</span>

    <span class="kw">def</span> handle_error(<span class="ot">self</span>, msg):
        content = <span class="ot">self</span>.Error_Page.<span class="dt">format</span>(path=<span class="ot">self</span>.path, msg=msg)
        <span class="ot">self</span>.send_content(content)</code></pre>

<p>This program works, but only if we don't look too closely. The problem is that it always returns a status code of 200, even when the page being requested doesn't exist. Yes, the page sent back in that case contains an error message, but since our browser can't read English, it doesn't know that the request actually failed. In order to make that clear, we need to modify <code>handle_error</code> and <code>send_content</code> as follows:</p>

<pre class="sourceCode python"><code class="sourceCode python">    <span class="co"># Handle unknown objects.</span>
    <span class="kw">def</span> handle_error(<span class="ot">self</span>, msg):
        content = <span class="ot">self</span>.Error_Page.<span class="dt">format</span>(path=<span class="ot">self</span>.path, msg=msg)
        <span class="ot">self</span>.send_content(content, <span class="dv">404</span>)

    <span class="co"># Send actual content.</span>
    <span class="kw">def</span> send_content(<span class="ot">self</span>, content, status=<span class="dv">200</span>):
        <span class="ot">self</span>.send_response(status)
        <span class="ot">self</span>.send_header(<span class="st">&quot;Content-type&quot;</span>, <span class="st">&quot;text/html&quot;</span>)
        <span class="ot">self</span>.send_header(<span class="st">&quot;Content-Length&quot;</span>, <span class="dt">str</span>(<span class="dt">len</span>(content)))
        <span class="ot">self</span>.end_headers()
        <span class="ot">self</span>.wfile.write(content)</code></pre>

<p>Note that we don't raise <code>ServerException</code> when a file can't be found, but generate an error page instead. A <code>ServerException</code> is meant to signal an internal error in the server code, i.e., something that <em>we</em> got wrong. The error page created by <code>handle_error</code>, on the other hand, appears when the <em>user</em> got something wrong, i.e., sent us the URL of a file that doesn't exist. <a href="#fn1" class="footnoteRef" id="fnref1"><sup>1</sup></a></p>

<h2 id="listing-directories">Listing Directories</h2>

<p>As our next step, we could teach the web server to display a listing of a directory's contents when the path in the URL is a directory rather than a file. We could even go one step further and have it look in that directory for an <code>index.html</code> file to display, and only show a listing of the directory's contents if that file is not present.</p>

<p>But building these rules into <code>do_GET</code> would be a mistake, since the resulting method would be a long tangle of <code>if</code> statements controlling special behaviors. The right solution is to step back and solve the general problem, which is figuring out what to do with a URL. Here's a rewrite of the <code>do_GET</code> method:</p>

<pre class="sourceCode python"><code class="sourceCode python">    <span class="kw">def</span> do_GET(<span class="ot">self</span>):
        <span class="kw">try</span>:

            <span class="co"># Figure out what exactly is being requested.</span>
            <span class="ot">self</span>.full_path = os.getcwd() + <span class="ot">self</span>.path

            <span class="co"># Figure out how to handle it.</span>
            <span class="kw">for</span> case in <span class="ot">self</span>.Cases:
                handler = case()
                <span class="kw">if</span> handler.test(<span class="ot">self</span>):
                    handler.act(<span class="ot">self</span>)
                    <span class="kw">break</span>

        <span class="co"># Handle errors.</span>
        <span class="kw">except</span> <span class="ot">Exception</span> <span class="ch">as</span> msg:
            <span class="ot">self</span>.handle_error(msg)</code></pre>

<p>The first step is the same: figure out the full path to the thing being requested. After that, though, the code looks quite different. Instead of a bunch of inline tests, this version loops over a set of cases stored in a list. Each case is an object with two methods: <code>test</code>, which tells us whether it's able to handle the request, and <code>act</code>, which actually takes some action. As soon as we find the right case, we let it handle the request and break out of the loop.</p>

<p>These three case classes reproduce the behavior of our previous server:</p>

<pre class="sourceCode python"><code class="sourceCode python"><span class="kw">class</span> case_no_file(<span class="dt">object</span>):
    <span class="co">&#39;&#39;&#39;File or directory does not exist.&#39;&#39;&#39;</span>

    <span class="kw">def</span> test(<span class="ot">self</span>, handler):
        <span class="kw">return</span> not os.path.exists(handler.full_path)

    <span class="kw">def</span> act(<span class="ot">self</span>, handler):
        <span class="kw">raise</span> ServerException(<span class="st">&quot;&#39;</span><span class="ot">{0}</span><span class="st">&#39; not found&quot;</span>.<span class="dt">format</span>(handler.path))


<span class="kw">class</span> case_existing_file(<span class="dt">object</span>):
    <span class="co">&#39;&#39;&#39;File exists.&#39;&#39;&#39;</span>

    <span class="kw">def</span> test(<span class="ot">self</span>, handler):
        <span class="kw">return</span> os.path.isfile(handler.full_path)

    <span class="kw">def</span> act(<span class="ot">self</span>, handler):
        handler.handle_file(handler.full_path)


<span class="kw">class</span> case_always_fail(<span class="dt">object</span>):
    <span class="co">&#39;&#39;&#39;Base case if nothing else worked.&#39;&#39;&#39;</span>

    <span class="kw">def</span> test(<span class="ot">self</span>, handler):
        <span class="kw">return</span> <span class="ot">True</span>

    <span class="kw">def</span> act(<span class="ot">self</span>, handler):
        <span class="kw">raise</span> ServerException(<span class="st">&quot;Unknown object &#39;</span><span class="ot">{0}</span><span class="st">&#39;&quot;</span>.<span class="dt">format</span>(handler.path))</code></pre>

<p>and here's how we construct the list of case handlers at the top of the <code>RequestHandler</code> class:</p>

<pre class="sourceCode python"><code class="sourceCode python"><span class="kw">class</span> RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
    <span class="co">&#39;&#39;&#39;</span>
<span class="co">    If the requested path maps to a file, that file is served.</span>
<span class="co">    If anything goes wrong, an error page is constructed.</span>
<span class="co">    &#39;&#39;&#39;</span>

    Cases = [case_no_file(),
             case_existing_file(),
             case_always_fail()]

    ...everything <span class="kw">else</span> <span class="ch">as</span> before...</code></pre>

<p>Now, on the surface this has made our server more complicated, not less: the file has grown from 74 lines to 99, and there's an extra level of indirection without any new functionality. The benefit comes when we go back to the task that started this chapter and try to teach our server to serve up the <code>index.html</code> page for a directory if there is one, and a listing of the directory if there isn't. The handler for the former is:</p>

<pre class="sourceCode python"><code class="sourceCode python"><span class="kw">class</span> case_directory_index_file(<span class="dt">object</span>):
    <span class="co">&#39;&#39;&#39;Serve index.html page for a directory.&#39;&#39;&#39;</span>

    <span class="kw">def</span> index_path(<span class="ot">self</span>, handler):
        <span class="kw">return</span> os.path.join(handler.full_path, <span class="st">&#39;index.html&#39;</span>)

    <span class="kw">def</span> test(<span class="ot">self</span>, handler):
        <span class="kw">return</span> os.path.isdir(handler.full_path) and \
               os.path.isfile(<span class="ot">self</span>.index_path(handler))

    <span class="kw">def</span> act(<span class="ot">self</span>, handler):
        handler.handle_file(<span class="ot">self</span>.index_path(handler))</code></pre>

<p>Here, the helper method <code>index_path</code> constructs the path to the <code>index.html</code> file; putting it in the case handler prevents clutter in the main <code>RequestHandler</code>. <code>test</code> checks whether the path is a directory containing an <code>index.html</code> page, and <code>act</code> asks the main request handler to serve that page.</p>

<p>The only change needed to <code>RequestHandler</code> is to add a <code>case_directory_index_file</code> object to our <code>Cases</code> list:</p>

<pre class="sourceCode python"><code class="sourceCode python">    Cases = [case_no_file(),
             case_existing_file(),
             case_directory_index_file(),
             case_always_fail()]</code></pre>

<p>What about directories that don't contain <code>index.html</code> pages? The test is the same as the one above with a <code>not</code> strategically inserted, but what about the <code>act</code> method? What should it do?</p>

<pre class="sourceCode python"><code class="sourceCode python"><span class="kw">class</span> case_directory_no_index_file(<span class="dt">object</span>):
    <span class="co">&#39;&#39;&#39;Serve listing for a directory without an index.html page.&#39;&#39;&#39;</span>

    <span class="kw">def</span> index_path(<span class="ot">self</span>, handler):
        <span class="kw">return</span> os.path.join(handler.full_path, <span class="st">&#39;index.html&#39;</span>)

    <span class="kw">def</span> test(<span class="ot">self</span>, handler):
        <span class="kw">return</span> os.path.isdir(handler.full_path) and \
               not os.path.isfile(<span class="ot">self</span>.index_path(handler))

    <span class="kw">def</span> act(<span class="ot">self</span>, handler):
        ???</code></pre>

<p>It seems we've backed ourselves into a corner. Logically, the <code>act</code> method should create and return the directory listing, but our existing code doesn't allow for that: <code>RequestHandler.do_GET</code> calls <code>act</code>, but doesn't expect or handle a return value from it. For now, let's add a method to <code>RequestHandler</code> to generate a directory listing, and call that from the case handler's <code>act</code>:</p>

<pre class="sourceCode python"><code class="sourceCode python"><span class="kw">class</span> case_directory_no_index_file(<span class="dt">object</span>):
    <span class="co">&#39;&#39;&#39;Serve listing for a directory without an index.html page.&#39;&#39;&#39;</span>

    <span class="co"># ...index_path and test as above...</span>

    <span class="kw">def</span> act(<span class="ot">self</span>, handler):
        handler.list_dir(handler.full_path)


<span class="kw">class</span> RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):

    <span class="co"># ...all the other code...</span>

    <span class="co"># How to display a directory listing.</span>
    Listing_Page = <span class="st">&#39;&#39;&#39;\</span>
<span class="st">        &lt;html&gt;</span>
<span class="st">        &lt;body&gt;</span>
<span class="st">        &lt;ul&gt;</span>
<span class="st">        </span><span class="ot">{0}</span>
<span class="st">        &lt;/ul&gt;</span>
<span class="st">        &lt;/body&gt;</span>
<span class="st">        &lt;/html&gt;</span>
<span class="st">        &#39;&#39;&#39;</span>

    <span class="kw">def</span> list_dir(<span class="ot">self</span>, full_path):
        <span class="kw">try</span>:
            entries = os.listdir(full_path)
            bullets = [<span class="st">&#39;&lt;li&gt;</span><span class="ot">{0}</span><span class="st">&lt;/li&gt;&#39;</span>.<span class="dt">format</span>(e) 
                <span class="kw">for</span> e in entries <span class="kw">if</span> not e.startswith(<span class="st">&#39;.&#39;</span>)]
            page = <span class="ot">self</span>.Listing_Page.<span class="dt">format</span>(<span class="st">&#39;</span><span class="ch">\n</span><span class="st">&#39;</span>.join(bullets))
            <span class="ot">self</span>.send_content(page)
        <span class="kw">except</span> <span class="ot">OSError</span> <span class="ch">as</span> msg:
            msg = <span class="st">&quot;&#39;</span><span class="ot">{0}</span><span class="st">&#39; cannot be listed: </span><span class="ot">{1}</span><span class="st">&quot;</span>.<span class="dt">format</span>(<span class="ot">self</span>.path, msg)
            <span class="ot">self</span>.handle_error(msg)</code></pre>

<h2 id="the-cgi-protocol">The CGI Protocol</h2>

<p>Of course, most people won't want to edit the source of their web server in order to add new functionality. To save them from having to do so, servers have always supported a mechanism called the Common Gateway Interface (CGI), which provides a standard way for a web server to run an external program in order to satisfy a request.</p>

<p>For example, suppose we want the server to be able to display the local time in an HTML page. We can do this in a standalone program with just a few lines of code:</p>

<pre class="sourceCode python"><code class="sourceCode python"><span class="ch">from</span> datetime <span class="ch">import</span> datetime
<span class="dt">print</span> <span class="st">&#39;&#39;&#39;\</span>
<span class="st">&lt;html&gt;</span>
<span class="st">&lt;body&gt;</span>
<span class="st">&lt;p&gt;Generated </span><span class="ot">{0}</span><span class="st">&lt;/p&gt;</span>
<span class="st">&lt;/body&gt;</span>
<span class="st">&lt;/html&gt;&#39;&#39;&#39;</span>.<span class="dt">format</span>(datetime.now())</code></pre>

<p>In order to get the web server to run this program for us, we add this case handler:</p>

<pre class="sourceCode python"><code class="sourceCode python"><span class="kw">class</span> case_cgi_file(<span class="dt">object</span>):
    <span class="co">&#39;&#39;&#39;Something runnable.&#39;&#39;&#39;</span>

    <span class="kw">def</span> test(<span class="ot">self</span>, handler):
        <span class="kw">return</span> os.path.isfile(handler.full_path) and \
               handler.full_path.endswith(<span class="st">&#39;.py&#39;</span>)

    <span class="kw">def</span> act(<span class="ot">self</span>, handler):
        handler.run_cgi(handler.full_path)</code></pre>

<p>The test is simple: does the file path end with <code>.py</code>? If so, <code>RequestHandler</code> runs this program.</p>

<pre class="sourceCode python"><code class="sourceCode python">    <span class="kw">def</span> run_cgi(<span class="ot">self</span>, full_path):
        cmd = <span class="st">&quot;python &quot;</span> + full_path
        child_stdin, child_stdout = os.popen2(cmd)
        child_stdin.close()
        data = child_stdout.read()
        child_stdout.close()
        <span class="ot">self</span>.send_content(data)</code></pre>

<p>This is horribly insecure: if someone knows the path to a Python file on our server, we're just letting them run it without worrying about what data it has access to, whether it might contain an infinite loop, or anything else.<a href="#fn2" class="footnoteRef" id="fnref2"><sup>2</sup></a></p>

<p>Sweeping that aside, the core idea is simple:</p>

<ol style="list-style-type: decimal">
<li>Run the program in a subprocess.</li>
<li>Capture whatever that subprocess sends to standard output.</li>
<li>Send that back to the client that made the request.</li>
</ol>

<p>The full CGI protocol is much richer than this—in particular, it allows for parameters in the URL, which the server passes into the program being run—but those details don't affect the overall architecture of the system...</p>

<p>...which is once again becoming rather tangled. <code>RequestHandler</code> initially had one method, <code>handle_file</code>, for dealing with content. We have now added two special cases in the form of <code>list_dir</code> and <code>run_cgi</code>. These three methods don't really belong where they are, since they're primarily used by others.</p>

<p>The fix is straightforward: create a parent class for all our case handlers, and move other methods to that class if (and only if) they are shared by two or more handlers. When we're done, the <code>RequestHandler</code> class looks like this:</p>

<pre class="sourceCode python"><code class="sourceCode python"><span class="kw">class</span> RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):

    Cases = [case_no_file(),
             case_cgi_file(),
             case_existing_file(),
             case_directory_index_file(),
             case_directory_no_index_file(),
             case_always_fail()]

    <span class="co"># How to display an error.</span>
    Error_Page = <span class="st">&quot;&quot;&quot;\</span>
<span class="st">        &lt;html&gt;</span>
<span class="st">        &lt;body&gt;</span>
<span class="st">        &lt;h1&gt;Error accessing </span><span class="ot">{path}</span><span class="st">&lt;/h1&gt;</span>
<span class="st">        &lt;p&gt;</span><span class="ot">{msg}</span><span class="st">&lt;/p&gt;</span>
<span class="st">        &lt;/body&gt;</span>
<span class="st">        &lt;/html&gt;</span>
<span class="st">        &quot;&quot;&quot;</span>

    <span class="co"># Classify and handle request.</span>
    <span class="kw">def</span> do_GET(<span class="ot">self</span>):
        <span class="kw">try</span>:

            <span class="co"># Figure out what exactly is being requested.</span>
            <span class="ot">self</span>.full_path = os.getcwd() + <span class="ot">self</span>.path

            <span class="co"># Figure out how to handle it.</span>
            <span class="kw">for</span> case in <span class="ot">self</span>.Cases:
                <span class="kw">if</span> case.test(<span class="ot">self</span>):
                    case.act(<span class="ot">self</span>)
                    <span class="kw">break</span>

        <span class="co"># Handle errors.</span>
        <span class="kw">except</span> <span class="ot">Exception</span> <span class="ch">as</span> msg:
            <span class="ot">self</span>.handle_error(msg)

    <span class="co"># Handle unknown objects.</span>
    <span class="kw">def</span> handle_error(<span class="ot">self</span>, msg):
        content = <span class="ot">self</span>.Error_Page.<span class="dt">format</span>(path=<span class="ot">self</span>.path, msg=msg)
        <span class="ot">self</span>.send_content(content, <span class="dv">404</span>)

    <span class="co"># Send actual content.</span>
    <span class="kw">def</span> send_content(<span class="ot">self</span>, content, status=<span class="dv">200</span>):
        <span class="ot">self</span>.send_response(status)
        <span class="ot">self</span>.send_header(<span class="st">&quot;Content-type&quot;</span>, <span class="st">&quot;text/html&quot;</span>)
        <span class="ot">self</span>.send_header(<span class="st">&quot;Content-Length&quot;</span>, <span class="dt">str</span>(<span class="dt">len</span>(content)))
        <span class="ot">self</span>.end_headers()
        <span class="ot">self</span>.wfile.write(content)</code></pre>

<p>while the parent class for our case handlers is:</p>

<pre class="sourceCode python"><code class="sourceCode python"><span class="kw">class</span> base_case(<span class="dt">object</span>):
    <span class="co">&#39;&#39;&#39;Parent for case handlers.&#39;&#39;&#39;</span>

    <span class="kw">def</span> handle_file(<span class="ot">self</span>, handler, full_path):
        <span class="kw">try</span>:
            <span class="kw">with</span> <span class="dt">open</span>(full_path, <span class="st">&#39;rb&#39;</span>) <span class="ch">as</span> reader:
                content = reader.read()
            handler.send_content(content)
        <span class="kw">except</span> <span class="ot">IOError</span> <span class="ch">as</span> msg:
            msg = <span class="st">&quot;&#39;</span><span class="ot">{0}</span><span class="st">&#39; cannot be read: </span><span class="ot">{1}</span><span class="st">&quot;</span>.<span class="dt">format</span>(full_path, msg)
            handler.handle_error(msg)

    <span class="kw">def</span> index_path(<span class="ot">self</span>, handler):
        <span class="kw">return</span> os.path.join(handler.full_path, <span class="st">&#39;index.html&#39;</span>)

    <span class="kw">def</span> test(<span class="ot">self</span>, handler):
        <span class="kw">assert</span> <span class="ot">False</span>, <span class="st">&#39;Not implemented.&#39;</span>

    <span class="kw">def</span> act(<span class="ot">self</span>, handler):
        <span class="kw">assert</span> <span class="ot">False</span>, <span class="st">&#39;Not implemented.&#39;</span></code></pre>

<p>and the handler for an existing file (just to pick an example at random) is:</p>

<pre class="sourceCode python"><code class="sourceCode python"><span class="kw">class</span> case_existing_file(base_case):
    <span class="co">&#39;&#39;&#39;File exists.&#39;&#39;&#39;</span>

    <span class="kw">def</span> test(<span class="ot">self</span>, handler):
        <span class="kw">return</span> os.path.isfile(handler.full_path)

    <span class="kw">def</span> act(<span class="ot">self</span>, handler):
        <span class="ot">self</span>.handle_file(handler, handler.full_path)</code></pre>

<h2 id="discussion">Discussion</h2>

<p>The differences between our original code and the refactored version reflect two important ideas. The first is to think of a class as a collection of related services. <code>RequestHandler</code> and <code>base_case</code> don't make decisions or take actions; they provide tools that other classes can use to do those things.</p>

<p>The second is extensibility: people can add new functionality to our web server either by writing an external CGI program, or by adding a case handler class. The latter does require a one-line change to <code>RequestHandler</code> (to insert the case handler in the <code>Cases</code> list), but we could get rid of that by having the web server read a configuration file and load handler classes from that. In both cases, they can ignore most lower-level details, just as the authors of the <code>BaseHTTPRequestHandler</code> class have allowed us to ignore the details of handling socket connections and parsing HTTP requests.</p>

<p>These ideas are generally useful; see if you can find ways to use them in your own projects.</p>

<div class="footnotes">
<ol>
<li id="fn1"><p>We're going to use <code>handle_error</code> several times throughout this chapter, including several cases where the status code <code>404</code> isn't appropriate. As you read on, try to think of how you would extend this program so that the status response code can be supplied easily in each case.<a href="#fnref1">↩</a></p></li>
<li id="fn2"><p>Our code also uses the <code>popen2</code> library function, which has been deprecated in favor of the <code>subprocess</code> module. However, <code>popen2</code> was the less distracting tool to use in this example.<a href="#fnref2">↩</a></p></li>
</ol>
</div>
  </body>
</html>
